Include metadata in numpy/polars cache fingerprints to prevent collisions#1616
Merged
Merged
Conversation
…ions hash_numpy_array now includes shape and dtype in the hash, preventing collisions between arrays with identical raw bytes but different semantics (e.g., shape=(6,) vs shape=(2,3)). hash_polars_dataframe now includes column names and dtypes (schema) in the hash, preventing collisions between DataFrames with identical cell values but different column schemas. Existing caches will simply miss (different hash = recomputation), not produce incorrect results. Reported-by: Dem0
elijahbenizzy
approved these changes
May 30, 2026
elijahbenizzy
left a comment
elijahbenizzy
left a comment
Contributor
There was a problem hiding this comment.
And closing out my duplicate
Contributor
Author
separate PR? would have to validate license, etc. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix hash collisions in the caching subsystem's fingerprinting for numpy arrays and polars DataFrames.
Problem
hash_numpy_arrayused onlyobj.tobytes(), which discards shape and dtype. Arrays with identical raw bytes but different shapes (e.g.,shape=(6,)vsshape=(2,3)) or different dtypes (e.g.,float32(1.0)vsint32(1065353216)) produced identical cache keys.hash_polars_dataframeused onlyobj.hash_rows(), which discards column names. DataFrames with identical cell values but different schemas produced identical cache keys.Both could cause the cache to silently return incorrect results from a previous execution.
Fix
hash_numpy_array: prependf"{obj.shape}:{obj.dtype}"to the bytes before hashinghash_polars_dataframe: include column names and dtypes (schema) alongside row hashesBackwards compatibility
This changes hash output for numpy arrays and polars DataFrames. Existing caches will miss (different hash = recomputation), not produce incorrect results. Users will see a one-time recomputation after upgrading but no manual cache clearing is needed.
Tests
Added tests verifying:
Reported-by: Dem0